MPI interface and tools in HySoP

HySoP is able to run on multi-core/multi-nodes architectures, with distributed memory systems and is based on Message Passing Interface (MPI) standard. For a standard usage of the software, you do not need a deep understanding of MPI standard, and even no understanding at all.

Anyway, this part is a reminder of a few things to know for a more advanced usage of HySoP parallel capabilities. You can skip most of it if you are at ease with mpi.

Glossary

HySoP is able to run on hybrid systems, multi-nodes, multi-cores, multi-gpu. Let us then start with some clarifications about the vocabulary used in this documentation, concerning those hybrid architectures.

cluster : a set of connected computers, all the available ressources for your simulation.

core : a processing unit with its own memory (cache)

socket or processor: a single computing component with two or more cores

node : a single computer inside a cluster.

An example of a cluster is shown below.

../_images/cluster_general.jpg

What is important to notice is that there are different memory areas (cache, RAM …), shared or not between cores, sockets …

MPI implementation

HySoP is based on mpi4py which is the Python package implementing the MPI standard.

References:

Once again, a glossary is required to clearly defined standard concepts used in this documentation, concerning MPI.

MPI process: an instance of a running program. When executing a python programme as:

mpirun -np 4 python yourscript.py

it means you launch 4 autonomous ‘processes’, each of them executing its own code from the instructions in yourscript.py. Each process executes in its own address space, its own memory, but has the ability to communicate with the others via mpi communication routines calls. Communicate means more or less to read or write some data from/to another process memory. The number of mpi processes is not supposed to correspond to the number of real physical cores available on the cluster but it’s certainly not a good idea to use a number of mpi processes greater than the number of available physical cores. The mapping between processes and cores (that may be anywhere on the cluster) is done by default by MPI but may be driven by user.

Communicator: a group (i.e. an ordered set) of MPI processes, inside a ‘context’ of communication. The context defines the properties of the communicator and how communications occur between the processes of the communicator. Mind that a message sent in one context cannot be received in another context.

Rank : integer id of a process, relative to a given communicator. Mind that the same process may have different ranks in two different communicators. Ranks always start at 0.

Task : a ‘job’ to be done by a set of processes. For example, consider the case where you need to solve an advection problem, i.e. find the values of a scalar field in a given velocity field and another problem (Navier-Stokes not to mention it) used to find the velocity field. Then, two tasks can be defined: one for the advection problem, attributed to one mpi process which may drive a GPU, and another task, attributed to all other processes running in parallel to solve Navier-Stokes.

HySoP and mpi

When running a simulation with hysop, a default communicator is used, including all the processes attributed at runtime (by mpirun command), namely main_comm. The rank of each process in main_comm is main_rank

>>> from hysop.core.mpi import main_size, main_rank
>>> print(f"I am process number {main_rank} among {main_size}")
I am process number 0 among 1

Running a script containing the previous lines with:

mpirun -np 4 python myscript.py

ended in

I am process number  1  among  4
I am process number  2  among  4
I am process number  3  among  4
I am process number  0  among  4

The mpi4py implementation is hidden into hysop.mpi package. To call any mpi4py function, try

>>> from hysop.core.mpi import MPI
>>> # example : get main communicator size:
>>> c = MPI.COMM_WORLD.Dup()

mpi parameters

To pass mpi-related arguments in many functions and classes in HySoP, a specific object is used, namely MPIParams.

with the following attributes:

  • comm : a communicator, default=main_comm

  • task_id : … id of a task, default=DEFAULT_TASK_ID

  • rank : current process rank in comm, default=main_rank

  • on_task : boolean, true if the task_id of the current process is task_id, default=True

Example

>>> from hysop.core.mpi import main_comm
>>> from hysop import MPIParams
>>> # default
>>> mp = MPIParams()
>>> # new sub-communicator:
>>> comm = main_comm.Split() # create a subcomm
>>> mp = MPIParams(comm)

Most of the time, you will not have to set this parameter and can rely on default behavior. It’s only useful when several tasks are required.

Communicator splitting, task creation and so on are domain’jobs and the process is detailed in Domains.